Step 5: Analyzing the Output

Output analysis deals with drawing inferences about the actual system based on the simulation output. When conducting simulation experiments, extreme caution should be used when interpreting the simulation results. Since the results of a simulation experiment are random (given the probabilistic nature of the inputs), an accurate measurement of the statistical significance of the output is necessary.

People doing simulation in academia are often accused of working with contrived and often oversimplified assumptions, yet are extremely careful about ensuring the statistical significance of the model results. Simulation practitioners in industry, on the other hand, are usually careful to obtain valid model data, only to ignore the statistical issues associated with simulation output. Maintaining a proper balance between establishing model validation and establishing the statistical significance of simulation output is an important part of achieving useful results.

The most valuable benefit from simulation is to gain insight, not necessarily to find absolute answers. With this in mind, one should be careful about getting too pedantic about the precision of simulation output. With more than 60 combined years of experience in doing simulation modeling, Conway, Maxwell and Worona (1986) caution that attaching a statistical significance to simulation output can create a delusion that the output results are either more or less significant than they really are. They emphasize the practical, intuitive reading of simulation results. Their guideline is "If you can't see it with the naked eye, forget it."

The goal of conducting experiments is not just to find out how well a particular system operates, but hopefully to gain enough insight to be able to improve the system. Unfortunately, simulation output rarely identifies causes of problems, but only reports the symptomatic behavior of problems. Bottleneck activities, for example, are usually identified by looking for locations or queues that are nearly always full which feed into one or more locations that are sometimes empty. Detecting the source of the bottleneck is sometimes a bit trickier than identifying the bottleneck itself. Bottlenecks may be caused by excessive operation times, prolonged delays due to the unavailability of resources, or an inordinate amount of downtime. The ability to draw correct inferences from the results is essential to making system improvements.